Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Restore receipts after recent fix #4248

Merged
merged 125 commits into from
May 24, 2021
Merged

Restore receipts after recent fix #4248

merged 125 commits into from
May 24, 2021

Conversation

Longarithm
Copy link
Member

@Longarithm Longarithm commented Apr 22, 2021

Currently rebased on #4274 to reuse MigrationData.

This PR has two goals:

  1. Re-introduce receipts to the chain, which were previously lost because of the bug in apply_chunks fixed here: Fix(chain): fix apply chunks #4228
  2. Provide a script allowing to verify that added receipts were actually lost. Because usual nodes clean old data, it can be launched only on mainnet archival node dumps.

Test plan

Check that

  • incorrect receipts don't pass verification and there are no still missing receipts: restored-receipts-verifier -> test_checking_differences
  • new receipts are correctly added to chain: the epoch is right, missing chunks don't affect the process, all receipts are added: process_blocks -> test_restoring_receipts_mainnet

Copy link
Collaborator

@bowenwang1996 bowenwang1996 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a description to the PR on what this PR does and what the test plan is

chain/chain/src/chain.rs Outdated Show resolved Hide resolved
chain/chain/src/chain.rs Outdated Show resolved Hide resolved
utils/restored-receipts-verifier/Cargo.toml Outdated Show resolved Hide resolved
utils/restored-receipts-verifier/src/main.rs Outdated Show resolved Hide resolved
utils/restored-receipts-verifier/src/main.rs Outdated Show resolved Hide resolved
utils/restored-receipts-verifier/src/main.rs Outdated Show resolved Hide resolved
utils/restored-receipts-verifier/src/main.rs Outdated Show resolved Hide resolved
utils/restored-receipts-verifier/src/main.rs Outdated Show resolved Hide resolved
utils/restored-receipts-verifier/src/main.rs Outdated Show resolved Hide resolved
utils/restored-receipts-verifier/src/main.rs Outdated Show resolved Hide resolved
chain/chain/src/chain.rs Outdated Show resolved Hide resolved
chain/chain/src/chain.rs Outdated Show resolved Hide resolved
chain/chain/src/chain.rs Outdated Show resolved Hide resolved
utils/restored-receipts-verifier/src/main.rs Outdated Show resolved Hide resolved
neard/src/config.rs Outdated Show resolved Hide resolved
neard/src/config.rs Outdated Show resolved Hide resolved
neard/src/config.rs Outdated Show resolved Hide resolved
chain/chain/src/chain.rs Outdated Show resolved Hide resolved
// True iff the current block is the first one in the chain with the current version
pub is_first_block_of_version: bool,
// True iff the current block containing chunk for some specific shard is the first one in the
// chain with the current version
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to be only true if first epoch of the current version have block with chunk (and also currently only for shard 0)

@@ -272,15 +282,29 @@ lazy_static_include::lazy_static_include_bytes! {
MAINNET_STORAGE_USAGE_DELTA => "res/storage_usage_delta.json",
}

#[cfg(feature = "protocol_feature_restore_receipts_after_fix")]
lazy_static_include::lazy_static_include_bytes! {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe swap this with next constant so that things related to single migration is consecutive

runtime_adapter.get_epoch_protocol_version(&prev_epoch_id)?;
Ok(protocol_version != prev_epoch_protocol_version)
}
Err(_) => Ok(true),
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks very suspicious to me. Have you accounted for all possible ways an attacker can maliciously send you some block that such get_prev_epoch_id_from_prev_block returns an error?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've tried to handle error coming from KeyValueRuntime.get_prev_epoch_id_from_prev_block but now found that my implementation was incorrect. Fixed

Comment on lines 36 to 37
Ok(chain_store.get_epoch_id_of_last_block_with_chunk(prev_block_hash, shard_id)?
!= runtime_adapter.get_epoch_id_from_prev_block(prev_block_hash)?)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use separate variables to avoid very long expressions like this

@Longarithm Longarithm merged commit c7d8fe8 into master May 24, 2021
@Longarithm Longarithm deleted the restore-receipts branch May 24, 2021 11:36
@frol
Copy link
Collaborator

frol commented May 25, 2021

@Longarithm @bowenwang1996 Will the restored receipts be stored in the storage (e.g. accessible via GetReceipt view client method)?

P.S. The context for this question is that we need to handle these on Indexer Framework side, and we recently bumped into local receipts that were delayed and had to hotfix the behavior in #4316.

/cc @khorolets

@bowenwang1996
Copy link
Collaborator

Will the restored receipts be stored in the storage

No. We only store receipts that are present in some chunk and here it is a special case.

@Longarithm
Copy link
Member Author

Longarithm commented Aug 21, 2021

Result of applying receipts to current mainnet state:

Commit used for testing: 6d2530b

Logs:

$ ./target/debug/state-viewer apply --height 41798999 --shard_id 0
Aug 21 00:17:33.763  INFO near: Opening store database at "/home/Aleksandr1/.near/data"
Aug 21 00:17:34.060  INFO runtime: Tracking shards: {0}
Pre-balance: b4a0c9f6f783bfcde1b1d4b75c85341c1c0d2e075f58d7a231601193d14ff91a 75182679016125035577525
Pre-balance: b828e1925025d7e12d1cf1268d6d9316c9972f85913bf302458a33516c3ebaef 75031270125311804664489
Pre-balance: tom.zest.near 2110354731541133588492954595
Receipts to restore len = 383
Incoming receipts len = 390
Processed: 390, Delayed: 0
Post-balance: b4a0c9f6f783bfcde1b1d4b75c85341c1c0d2e075f58d7a231601193d14ff91a 77427050088811915307435
Post-balance: b828e1925025d7e12d1cf1268d6d9316c9972f85913bf302458a33516c3ebaef 77315256944165011573409
Post-balance: tom.zest.near 2110455899816030404631797163
apply chunk for shard 0 at height 41798999, resulting chunk extra ChunkExtraV1 { state_root: `3bGmsULynNUdN1jye64FoD1c2SdNg94q6tCAuSrhhKyy`, outcome_root: `Gsn5v5Aopgrj6bLXjnMDv7H5CAjMtW4R2NVqCWfKsnZx`, validator_proposals: [], gas_used: 267292638915101, gas_limit: 1000000000000000, balance_burnt: 22938360794841900000000 }
Existing chunk extra: ChunkExtraV1 { state_root: `C9FWq6sHK2QGrvyux5LQQA8fTZXkyYtagJdT8ZXLL1gr`, outcome_root: `J9rwHeE6pv3oRyYWMVPtKgrLTxNS2JyPB68rFrVwj1Ae`, validator_proposals: [], gas_used: 52966524587327, gas_limit: 1000000000000000, balance_burnt: 4001453174034300000000 }

"Processed: 390, Delayed: 0" shows that all restored receipts were applied, and exit code 0 shows that no issues were happened.

Some research of accounts balance.
Compute post-balance minus pre-balance minus all deposits made in restored receipts, we expect to get 0.

b4a0c9f6f783bfcde1b1d4b75c85341c1c0d2e075f58d7a231601193d14ff91a:
mainnet_restored_receipts.json contains one refund for it;
>>> 77427050088811915307435-75182679016125035577525-2244371072686879729910 
0

b828e1925025d7e12d1cf1268d6d9316c9972f85913bf302458a33516c3ebaef:
mainnet_restored_receipts.json contains one refund for it;
>>> 77315256944165011573409-75031270125311804664489-2283986818853206908920 
0

tom.zest.near
mainnet_restored_receipts.json contains two refunds for it;
>>> 2110455899816030404631797163-2110354731541133588492954595-10000000000000000000000-22561702411076162935108
68606572485739975907460

But receipts in considered block also affect this account. There are two such ones, and

>>> 68606572485739975907460-18606572485739975907460-50000000000000000000000
0

Same thing was checked for two previous blocks.

matklad added a commit to matklad/nearcore that referenced this pull request May 29, 2022
This was one-off thing we needed a while ago. We no longer use it, so
there's no need to support it. The code can always be restored from the
git history.

cc near#4248
near-bulldozer bot pushed a commit that referenced this pull request May 31, 2022
This was one-off thing we needed a while ago. We no longer use it, so
there's no need to support it. The code can always be restored from the
git history.

cc #4248
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants